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PROBLEM TO BE SOLVED: To provide an image 
synthesizing apparatus providing panoramic images of high 
accuracy even in the absence of sufficient matching 
between point features due to small amount of textures on 
images. 

SOLUTION: The image synthesizing apparatus for 
synthesizing two overlapping images includes a first means 
for extracting a linear segment of each image from the 
overlapping parts of the two image, a second means for 
obtaining the matching of the linear segments between both 
images, a third means for calculating a geometrical 
conversion parameter between both images, and a fourth 
means for synthesizing both images using the calculated 
geometrical conversion parameter. 
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* NOTICES * 

Japan Patent Office is not responsible for any 
damages caused by the use of this translation. 

1. This document has been translated by computer. So the translation may not reflect 
the original precisely. 

2 **** s hows the word which can not be translated. 
3.1n the drawings, any words are not translated. 



CLAIMS 



[Claim(s)] 

[Claim 1] In an image synthesizer unit which compounds an image of two sheets which 
has a lap portion from a lap portion of an image of two sheets 1st means to extract a 
straight line segment of each image, 2nd means to gain matching of a straight line 
segment between both images, An image synthesizer unit characterized by having 4th 
means to compound both images from matching of a straight line segment between both 
gained images using 3rd means to compute a geometric conversion parameter between 
both images, and a computed geometric conversion parameter. 
[Claim 2] The 2nd means is an image synthesizer unit [ equipped with a means to 
predict a coordinate in the 2nd image of another side from a coordinate of an endpoint of 
a straight line segment with one 1st image, and a means to search for a predicted 
straight line segment of a coordinate value of an endpoint in the 2nd image, and the 2nd 
image with whenever / highest / matching ] according to claim 1 using a means to 
presume an optical flow between both images, and an obtained optical flow presumption 
result. 

[Claim 3] In an image composition method which compounds an image of two sheets 
which has a lap portion from a lap portion of an image of two sheets The 1st step which 
extracts a straight line segment of each image, the 2nd step which gains matching of a 
straight line segment between both images, An image composition method 
characterized by having the 4th step which compounds both images from matching of a 
straight line segment between both gained images using the 3rd step which computes a 
geometric conversion parameter between both images, and a computed geometric 
conversion parameter. 

[Claim 4] The 2nd step is the image composition method [ equipped with a step which 
predicts a coordinate in the 2nd image of another side from a coordinate of an endpoint 
of a straight line segment with one 1st image, and a step which searches for a predicted 
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straight line segment of a coordinate value of an endpoint in the 2nd image, and the 2nd 
image with whenever / highest / matching ] according to claim 3 using a step which 
presumes an optical flow between both images, and an obtained optical flow 
presumption result. 

[Claim 5] It is the record medium which recorded an image composition processing 
program which compounds an image of two sheets which has a lap portion and in which 
computer reading is possible. The 1st step which extracts a straight line segment of 
each image from a lap portion of an image of two sheets, From matching of a straight 
line segment between both images that gain matching of a straight line segment 
between both images and that were gained the 2nd step A record medium which a 
computer was made to perform the 4th step which compounds both images using the 
3rd step which computes a geometric conversion parameter between both images, and a 
computed geometric conversion parameter, and recorded an image composition 
processing program of a **** sake and in which computer reading is possible. 
[Claim 6] A step the 2nd step presumes an optical flow between both images to be, A 
step which predicts a coordinate in the 2nd image of another side from a coordinate of 
an endpoint of a straight line segment with one 1st image using an obtained optical flow 
presumption result, And a record medium which recorded an image composition 
processing program [ equipped with a step which searches for a predicted straight line 
segment of a coordinate value of an endpoint in the 2nd image, and the 2nd image with 
whenever / highest / matching ] according to claim 5 and in which computer reading is 
possible. 



DETAILED DESCRIPTION 



[Detailed Description of the Invention] 
[0001] 

[The technical field to which invention belongs] This invention relates to the record 
medium which recorded the image synthesizer unit, the image composition method, and 
the image composition program. 
[0002] 

[Description of the Prior Art] [l] Explanation about the calculation method of the 
conventional optical flow [0003] An optical flow is calculated from the image of two 
sheets, and the technology of performing alignment between the images of two sheets is 
known based on the obtained optical flow. The calculation method of the conventional 
OPUTE cull flow is explained. 
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[0004] (1) From the Lucas-Kanade method former, many technique of calculating the 
speed field (optical flow) of the appearance of the movement body in a dynamic image is 
proposed. Lucas-Kanade which is a partial gradient method especially law is one of 
the best technique. The reason is that processing has ease [ a high speed and mounting ] 
and a result has reliability. 

[0005] About the details of the Lucas-Kanade method, it is reference. : Refer to B.Lucas 
and T.Kanade and "An Iterative Image Registration Technique with an Application to 
Stereo Vision" In Seventh International Joint Conference on Artificial Intelligence 
(IJCAI-81), pp.674-979, and 1981. 

[0006] Below, the outline of the Lucas-Kanade method is described. When the shade 
pattern I (x y, t) of image coordinate p= (x y) of a certain time of day t moves after a 
certain minute time amount (deltat), keeping the shade distribution constant on a 
coordinate (x+delta x, y+deltay), the following optical flow restricted type 1 is realized. 
[0007] 

[Equation l] 

[0008] Since the number of unknown parameters is two pieces in order to calculate an 
optical flow {v=(deltax/deltat, deltay/deltat) = (u, v)} by the two-dimensional image, a 
piece restricted type is already required. Lucas Kanade (****) In the partial field of the 
same body, it was assumed that it had the same optical flow. 

[0009] For example, the square error E of a shade pattern to minimize in the partial 
field omega on an image supposing an optical flow is fixed is 10. (p) =1 (x y, t), II =(p+v) I 
(x+u, y+v, t+deltat) 

If it rewrites, it can define by the degree type 2. 
[0010] 

[Equation 2] 

[0011] Since v can disregard the secondary more than term of the Taylor expansion here 

when very small, the relation of the degree type 3 is realized. 

[0012] 

[Equation 3] 

[0013] Here, g (p) is II. It is the primary differential of (p). 

[0014] Since it is a time of the differential value over v of E being 0 that Error E 

becomes min, the relation of the degree type 4 is realized. 

[0015] 
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[Equation 4] 

[0016] Therefore, an optical flow v is called for by the degree type 5. 
[0017] 

[Equation 5] 

[0018] Furthermore, it can ask to be shown in the degree type 6 with a sufficient 

precision by the Newton RAFUSON-iterative operation. 

[0019] 

[Equation 6] 

[0020] (2) Since good initial value is required for the biggest trouble of a gradient 
method including the hierarchical presuming method Lucas-Kanade method, it is 
inapplicable to a big motion. Then, the method of creating and solving from the former 
the image with which the resolution of several step time differs in a pyramid layered 
structure mold is proposed. 

[0021] This creates first the image with which several steps of resolution of each image 
differs beforehand from the continuous image of two sheets. Next, a rough optical flow is 
calculated between images with the lowest resolution. And it refers to this result and a 
more precise optical flow is calculated between images with high one-step resolution. 
This processing is successively repeated between images with the highest resolution. 
[0022] drawing 4 - a subject-copy image ■- in drawing 3 , drawing 2 shows an image 
with resolution lower than the low resolution picture of drawing 3 , and drawing 1 
shows the image with resolution lower than the low resolution picture of drawing 2 for 
the image with resolution lower than the subject-copy image of drawing 4 , respectively. 
In drawing 1 - drawing 4 , S shows one patch. 

[0023] An optical flow is gradually called for in order of the image (a hierarchy's 1 
image) of drawing 1 , the image (a hierarchy's 2 image) of drawing 2 , the image (a 
hierarchy's 3 image) of drawing 3 , and the image (a hierarchy's 4 image) of drawing 4 . 
In drawing 1 - drawing 4 , the arrow head shows the optical flow vector searched for for 
every patch. 

[0024] However, by the real image, a trouble here has few fields including sufficient 
pattern (texture), and is not to obtain a reliable optical flow. 

[0025] [2] Explanation about the optical flow calculation method which these people 
developed. 

[0026] The optical flow calculation method which these people developed creates the 
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image with which the resolution of several step time differs in a pyramid hierarchical 
type, and is premised on hierarchical presumption which calculates an optical flow 
gradually, the count method of an optical flow - Lucas-Kanade - gradient methods, 
such as law, are followed. That is, it is premised on the optical flow presuming method 
by the layered-structure-ized gradient method. Here, the Lucas-Kanade method is used 
as a gradient method. 

[0027] Lucas-Kanade which layered-structure ized the feature of the optical flow 
presumption method which these people developed - it is in complementing the optical 
flow obtained in each phase of the optical flow presuming method by law by expansion 
processing. Hereafter, this is explained in detail. 

[0028] One of the advantages of the Lucas-Kanade method is that a trace result has 
reliability. Tomasi Kanade The trace possibility of a certain field showed that it was 
computable from a differential image as follows (C. Tomasi and T.Kanade and "Shape 
and Motion from Image Streams - - a Factorization method-Part 3 Detection and Tracking 
of Point Features" CMU CS -91-132, Carnegie Mellon University, 1991.). 
[0029] From the coefficient matrix G of 2x2 of the degree type 7 which has the square of 
the perpendicular and horizontal differential of a certain field image omega in an 
element, the trace possibility of the field can be determined by calculating the 
characteristic value. 
[0030] 

[Equation 7] 

[0031] When the characteristic value of both of this matrix G is large, that field has 
change in the rectangular direction, and positioning of a meaning is possible for it. 
Therefore, characteristic value lambdamin of the smaller one The reliability gamma of a 
trace result can be acquired from the shade remainder E between the fields after a trace 
by the degree type 8. 
[0032] 

[Equation 8] 

[0033] this invention persons developed the method of interpolating the field where 
reliability is low using the high result of reliability within the same hierarchy of an 
optical flow, this " a single step " the result in a coarse hierarchy is used only for the 
initial value of a trace, and nothing uses it for the result of the hierarchy of a present 
stage who pays his attention. Instead, the optical flow of a field with few textures 
assumes that it has a value near the optical flow of the perimeter, and complements a 
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flow field by mol follow G processing. 

[0034] The situation of expansion processing of a flow vector is shown in drawing 5 . 
[0035] The left figure expresses the map of the reliability of a flow vector with a shade. 
Here, the more it is black, the more suppose that reliability is high. 
[0036] First, threshold processing of the acquired flow is carried out. Since the 
reliability of a result is low, threshold processing of the white portion is carried out. 
[0037] Next, the stopgap processing by the mol follow G operation in a binary image is 
imitated, and expansion processing of a result is performed as follows in a flow field. 
From the about four flow vector, the flow vector u of some fields i and j (i, j) performs 
weighting according to reliability gamma, and can calculate it like the degree type 9. 
[0038] 

[Equation 9] 

[0039] This processing is repeated until the field where all the reliability by which 
threshold processing was carried out is low is buried. This complement processing is 
performed in each hierarchy. In addition, you may make it compute the flow vector u of 
some fields i and j (i, j) by performing weighting according to reliability gamma from the 
about eight flow vector. 

[0040] Drawing 6 (a) shows the optical flow by which threshold processing was carried 
out to a certain hierarchy's image, and drawing 6 (b) shows the optical flow after a 
complement. In drawing 6 (a), an arrow head is the optical flow vector judged as 
reliability being high by threshold processing, and x mark shows the portion made low 
[ reliability ]. 

[0041] [3] explanation of the generation method of the conventional panorama image - 
although there is a method using the omnidirection sensor which used the fish-eye lens 
and the mirror among the methods of gaining the image of a large visual field, it is 
expensive and there is a problem that resolution is low. 

[0042] The technique of, sticking and setting two or more images picturized with the 
usual camera on the other hand, and compounding a panorama image is proposed. As 
this kind of the image composition method, the method which this applicant for a patent 
developed is learned (refer to JP,11"339021,A). By the image composition method which 
this applicant for a patent has already developed, a panorama image is compounded by 
extracting the characteristic small rectangle image (point feature) of an image, 
matching these focus between the images of two sheets, and computing the geometric 
conversion parameter between the images of two sheets from those correspondence 
relation. 
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[0043] however - the case where the point features, such as a wall of the monochrome of 
a building, are not fully acquired by the above-mentioned conventional method - 
precision there is a problem that it is difficult to compute a geometric conversion 
parameter highly. 
[0044] 

[Problem(s) to be Solved by the Invention] As for this invention, an image has few 
patterns (texture), and even when correspondence of sufficient point feature is not 
obtained, it aims at offering the record medium which recorded the image synthesizer 
unit and the image composition method that a panorama image with a high precision is 
obtained, and the image composition program. 
[0045] 

[Means for Solving the Problem] In an image synthesizer unit with which an image 
synthesizer unit by this invention compounds an image of two sheets which has a lap 
portion 1st means to extract a straight line segment of each image from a lap portion of 
an image of two sheets, From matching of a straight line segment between both images 
that gain matching of a straight line segment between both images and that were 
gained the 2nd means It is characterized by having 4th means to compound both images, 
using 3rd means to compute a geometric conversion parameter between both images, 
and a computed geometric conversion parameter. 

[0046] A thing equipped with a means to predict a coordinate in the 2nd image of 
another side from a coordinate of an endpoint of a straight line segment which has one 
1st image as the 2nd means, for example using a means to presume an optical flow 
between both images, and an obtained optical flow presumption result, and a means to 
search for a predicted straight line segment of a coordinate value of an endpoint in the 
2nd image and the 2nd image with whenever [ matching / highest ] is used. 
[0047] In an image composition method that an image composition method by this 
invention compounds an image of two sheets which has a lap portion The 1st step which 
extracts a straight line segment of each image from a lap portion of an image of two 
sheets, From matching of a straight line segment between both images that gain 
matching of a straight line segment between both images and that were gained the 2nd 
step It is characterized by having the 4th step which compounds both images using the 
3rd step which computes a geometric conversion parameter between both images, and a 
computed geometric conversion parameter. 

[0048] The 2nd step is equipped with a step which predicts a coordinate in the 2nd 
image of another side from a coordinate of an endpoint of a straight line segment with 
one 1st image, and a step which searches for a predicted straight line segment of a 
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coordinate value of an endpoint in the 2nd image, and the 2nd image with whenever 
[ matching / highest ] using a step which presumes an optical flow for example, between 
both images, and an obtained optical flow presumption result. 

[0049] A record medium which recorded an image composition processing program by 
this invention and in which computer reading is possible The 1st step which extracts a 
straight line segment of each image from a lap portion of an image of two sheets, From 
matching of a straight line segment between both images that gain matching of a 
straight line segment between both images and that were gained the 2nd step It is 
characterized by making a computer perform the 4th step which compounds both 
images using the 3rd step which computes a geometric conversion parameter between 
both images, and a computed geometric conversion parameter, and recording an image 
composition processing program of a **** sake. 

[0050] The 2nd step is equipped with a step which predicts a coordinate in the 2nd 
image of another side from a coordinate of an endpoint of a straight line segment with 
one 1st image, and a step which searches for a predicted straight line segment of a 
coordinate value of an endpoint in the 2nd image, and the 2nd image with whenever 
[ matching / highest ] using a step which presumes an optical flow for example, between 
both images, and an obtained optical flow presumption result. 
[0051] 

[Embodiment of the Invention] Hereafter, the gestalt of implementation of this 
invention is explained. 

[0052] [2] Explanation of a panorama image synthesizer unit [0053] Drawing 7 shows 
the configuration of a panorama image synthesizer unit. 

[0054] A panorama image synthesizer unit is realized by the personal computer. The 
display 21, the mouse 22, and the keyboard 23 are connected to the personal computer 
10. The personal computer 10 is equipped with the drive (disk drive) 14 of CPU11, 
memory 12, a hard disk 13, and a removable disk like CD-ROM. 

[0055] a hard disk 13 - OS (operating system) etc. - others " the panorama image 
composition processing program is stored. A panorama image composition processing 
program is installed on a hard disk 13 using CD-ROM20 in which it was stored. 
Moreover, two or more images picturized with the digital camera shall be beforehand 
stored in a hard disk 13. 

[0056] [2] Explanation of the panorama image composition processing performed by 
CPU 11 when a panorama image composition program is made to start [0057] Drawing 8 
shows the panorama image composition procedure performed by CPU11. 
[0058] Here, the image of two sheets on expedient of explanation is compounded, and 
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the case where a panorama image is generated is explained. 

[0059] First, the image of two sheets which should be compounded is read into memory 
12 (step 1). Suppose that the image (the 1st image Al and the 2nd image A2) of two 
sheets of explanation which have the lap section for convenience here was read into 
memory 12. 

[0060] Extract processing of the lap section of the 1st image Al and the 2nd image A2 is 
performed (step 2). SSD which describes extract processing of this lap section below - it 
is carried out based on law (Sum of Squared Difference) or a normalization 
cross-correlation method. In addition, like a video image, an input image can skip this 
step, when an inter-frame difference is small. 

[0061] (a) SSD - the explanation SSD of law - the images Al and A2 of two sheets 
which should extract the lap section first in law - each the image II with resolution 
lower than a subject-copy image, and 12 It is generated. The low resolution picture II of 
two sheets, and 12 It is asked using the square error E per pixel for the lap portion 
omega (size: MxN) to show in the degree type 10. The movement magnitude between 
images (d) is made to change in the possible range, and the lap section is extracted for E 
from the smallest movement magnitude (d). 
[0062] 

[Equation 10] 

[0063] (b) the images Al and A2 of two sheets which should extract the lap section first 
by the explanation normalization cross-correlation method of a normalization 
cross-correlation method each " the image II with resolution lower than a 
subject-copy image, and 12 It is generated. The low resolution picture II of two sheets, 
and 12 It is asked using the normalization cross correlation function C for the lap 
portion omega (size*. MxN) to show in the degree type 11. The movement magnitude 
between images (d) is made to change in the possible range, and the lap section is 
extracted from movement magnitude (d) with biggest C. 
[0064] 

[Equation 11] 

[0065] In a formula 11, II- and I2~ are the averages of the gray level of each of each 
image in the lap section of both images when the 1st image is made to fix and only d 
moves the 2nd image. Moreover, mhol and mho2 The 1st image II It is made to fix and 
is the 2nd image 12. It is distribution of the gray level of each of each image in the lap 
section of both images when only d makes it move. 
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[0066] Next, line feature-extraction processing is performed (step 3). That is, the line 
feature of each images Al and A2 is extracted from a lap portion with the 2nd image A2 
in the 1st image Al. 

[0067] A general method is used for this. For example, Canny After edge extract 
processing (J.F.Canny "Finding edges and lines in images" MIT Technical Report, 
AI-TR -720, 1983) extracts an edge, the acquired edge pixel train is divided for every 
straight line. A linear equation is applied using the edge pixel divided for every straight 
line. Here, in addition to a linear equation and the coordinate of an endpoint, the 
information of change of an edge on whether one of linear is bright, i.e., the direction, is 
used for the attribute which each straight line segment has. 

[0068] Next, the line feature matching processing is performed (step 4). The straight 
line segment extracted by each images Al and A2 is matched between an image Al and 
A2. 

[0069] Here, the straight line segment extracted by each images Al and A2 is matched 
between an image Al and A2 by the method indicated by reference (Naoki Chiba, **** 
Takeo, the "line feature trace robust to a way piece or contiguity arrangement", 
Institute of Electronics, Information and Communication Engineers paper magazine 
Vol J81-D-II No.8, 1998). The line feature matching processing consists of the following 
three steps. 

[0070] (1) The 1st step [ 1st ] of a step is a step which presumes the optical flow between 
images. At this step, an optical flow is presumed for every patch based on the optical 
flow presumption method (refer to [2] of the column of explanation of the conventional 
technology) which these people developed. Under the present circumstances, based on 
reliability, the result of the patch with reliability lower than a threshold is thrown away, 
and it complements by expansion processing using the result of a high patch of 
surrounding reliability. Moreover, these processings are given to an 
of-condensation-and-rarefaction analysis target using the pyramid image which 
changed and created resolution. 

[0071] (2) The 2nd step [ 2nd ] of a step is a step which computes the coordinate in the 
2nd image using the obtained optical flow presumption result from the coordinate of the 
endpoint of a straight line segment with the 1st image Al. The flow vector over the 
coordinate of an endpoint is computed in the unit below a pixel by congruence linear 
interpolation using the optical flow presumption result of a nearby patch. 
[0072] (3) It is the step which searches for the straight line segment of the 2nd image 
which matches the coordinate value in the 2nd predicted image most at the 3rd step 
[ 3rd] of a step. 
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[0073] Here, the 3rd step is searched based on the similarity shown below. It is Li about 
the location of the straight line segment in the 2nd image of a straight line segment 
with the 1st image calculated here using optical flow presumption between images as 
shown in drawing 9 . It is Lj about the straight line segment to which it carries out and 
its attention is paid in the straight line segment of the 2nd image. It carries out. And 
both [ these ] the straight line segment Li, the distance D between Lj(s), and the rate P 
of a lap are defined. At this time, the median line calculated from the equation between 
2 straight-line segments is used. First, distance D is made into the sum of the distance 
from each straight line segment to the median line, and is shown by the degree type 12. 
[0074] 

[Equation 12] 

[0075] Moreover, the rate P of a lap projects each straight line segment on the median 
line, and is shown by the degree type 13 from the length Pc in which both those 
projected straight line segments exist in common, and the length Pw in which one of 
straight lines exist. 
[0076] 

[Equation 13] 

[0077] And Similarity S is shown by the degree type 14 from these distance D and rates 

P of a lap. 

[0078] 

[Equation 14] 

[0079] However, kd And kp It is a constant and sets experimentally. 

[0080] A straight line segment with the highest similarity is chosen as what matched 

most using this similarity S. 

[0081] Next, a geometric conversion parameter is computed (step 5). 
[0082] The geometric conversion parameter between images is computed from 
correspondence of the straight line segment gained at the above-mentioned step 4. Here, 
plane projective transformation is explained to an example as a geometric conversion 
model. First, a straight line segment is expressed by homogeneous coordinates from the 
attribute of a straight line segment (refer to O.Faugeras and "Three-Dimensional 
Computer Vision : A Geometric Viewpoint" MIT Press and 1993). 

[0083] Here, between the straight line segment 11 of the 1st image expressed by 
homogeneous coordinates, and the straight line segment 12 of the 2nd corresponding 
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image, it is the geometric conversion HL. It uses and the relation expressed with the 

degree type 15 is realized. 

[0084] 

[Equation 15] 

[0085] Here, it is Matrix HL. It is computable using correspondence of 4 or more sets of 
straight line segments, moreover, geometric transformation matrix HP of the coordinate 
between 2 images This HL from - computable (refer to O.Faugeras and 
"Three-Dimensional Computer Vision : A Geometric Viewpoint" MIT Press and 1993) by 
the degree type 16. 
[0086] 

[Equation 16] 

[0087] Here, it is t. The transpose of a matrix and A-l show the inverse matrix of Matrix 
A. 

[0088] Next, image composition processing is performed (step 6). 

[0089] The 1st image Al and the 2nd image A2, and geometric transformation matrix 
HP computed at step 5 It is based and the 1st image Al and the 2nd image A2 are 
compounded. 
[0090] 

[Effect of the Invention] According to this invention, an image has few patterns 
(texture), and even when correspondence of sufficient point feature is not obtained, a 
panorama image with a high precision comes to be obtained. 

DESCRIPTION OF DRAWINGS 

[Brief Description of the Drawings] 

fDrawing l] Drawing 1 is drawing for explaining the hierarchical presuming method, 
and is the mimetic diagram showing a hierarchy's 1 image. 

[Drawing 2] Drawing 2 is drawing for explaining the hierarchical presuming method, 
and is the mimetic diagram showing a hierarchy's 2 image. 

[Drawing 3] Drawing 3 is drawing for explaining the hierarchical presuming method, 
and is the mimetic diagram showing a hierarchy's 3 image. 

[Drawing 41 Drawing 4 is drawing for explaining the hierarchical presuming method, 
and is the mimetic diagram showing a hierarchy's 4 image. 

[Drawing 51 Drawing 5 is a mimetic diagram for explaining the expansion processing 
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performed in the optical flow presumption method adopted in the example. 
[Drawing 6l Drawing 6 (a) is the mimetic diagram showing the example of the optical 
flow by which threshold processing was carried out to a certain hierarchy's image, and 
drawing 6 (b) is the mimetic diagram showing the optical flow after a complement. 
[Drawing 7] Drawing 7 is the block diagram showing the configuration of a panorama 
image synthesizer unit. 

[Drawing 8] Drawing 8 is a flow chart which shows panorama image composition 
procedure. 

[Drawing 9] It is a mimetic diagram for explaining the line feature matching processing. 
[Description of Notations] 

10 Personal Computer 

11 CPU 

12 Memory 

13 Hard Disk 

14 Drive of Removable Disk 

21 Display 

22 Mouse 

23 Keyboard 
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